Topic 2 – Exploratory Data Analysis (EDA)
ENVX1002 Introduction to Statistical Methods
Dec 2024
Presentation of data
Tables: Experimental data
Presentation of data
Tables: Observational data
Population versus Sample
Before we go calculate averages, we need to think about the difference between population and sample
- We take a sample from a larger population
- What information does the sample give about the population and how reliable is that information?
bakhtiarzein - https://stock.adobe.com/
Motivating example
- Sequestered soil carbon is worth $35/tonne if measured (1 Tonne of Carbon = 1 Australian Carbon Credit Unit = $AU35 See Clean Energy Regulator)
- It costs $100 to collect and analyse one soil sample for soil carbon
- The farmer needs an estimate of carbon stored on the property.
- How many samples are needed to give a good estimate of carbon on the property? Is it worth measuring soil carbon for a land holder?
Motivating example
Google earth image with farm and soil-landscape boundaries
Sigma notation
- \Sigma, is the greek capital letter called sigma, refers to the sum
- It is a convenient way to represent long sums
\Sigma_{i=1}^n=x_1+x_2+x_3+...+x_n
- sum(c(48, 56, 78, 86, 90, 271))
[1] 629
- =SUM(A1:A6)
Centre: Arithmetic mean
- Population mean (\mu): sum of all values of a variable divided by the number of objects in the population;
\mu = \frac{\sum_{i=1}^{N} y_i}{N}
- Sample mean (\overline{y}) is based on a subset of n objects from a population of size N
\overline{y} = \frac{\sum_{i=1}^{n} y_i}{n}
- mean(c(48, 56, 78, 86, 90, 271))
[1] 104.8333 - =AVERAGE(A1:A6)
Centre: Median
- Median is the middle number of a set of ordered observations
Population median: M=\left(\frac{N+1}{2}\right)th sorted value
Sample median: \tilde{y}=\left(\frac{n+1}{2}\right)th sorted value
- median(c(48, 56, 78, 86, 90, 271))
[1] 82 - =MEDIAN(A1:A6)
Centre: Mode
Mode is the most commonly occurring number in a set of observations
-
[1] 4 - =MODE.SNGL(A1:A7)
Spread: Range
- Difference between largest and smallest observations in a group of data
- Note that we also refer to spread as measures of dispersion
- max(c(48, 56, 78, 86, 90, 271)) - min(c(48, 56, 78, 86, 90, 271))
[1] 223 - =MAX(A1:A6) - MAX(A1:A6)
Spread: Inter-quartile range (IQR)
- IQR = Q_3 - Q_1
Source: Nicholas (1999)
Spread: Inter-quartile range (IQR)
Quartiles
- quantile(c(48, 56, 78, 86, 90, 271))
0% 25% 50% 75% 100%
48.0 61.5 82.0 89.0 271.0 - =QUARTILE.INC(A1:A6, 1) - first quartile
Spread: Inter-quartile range (IQR)
IQR
- IQR(c(48, 56, 78, 86, 90, 271))
[1] 27.5 - =QUARTILE.INC(A1:A6, 3)-QUARTILE.INC(A1:A6, 1) - third quartile - first quartile
Spread: Variance
- Describes variability around the arithmetic mean
Population variance: \sigma^2 = \frac{\sum_{i=1}^{N}(y_i - \mu)^2}{N}
Sample variance: s^2 = \frac{\sum_{i=1}^{n}(y_i - \overline{y})^2}{n-1}
- var(c(48, 56, 78, 86, 90, 271))
[1] 6904.167 - =VAR.S(A1:A6)
Spread: Standard deviation
- R’s sd function always calculates the sample standard deviation
- The denominator of sample standard is n−1 (Bessel’s correction) this is an important concept in statistics. A key property is that it gives a more accurate estimate of the population variance and standard deviation when working with a sample.
- sd(c(48, 56, 78, 86, 90, 271))
[1] 83.09132 - =STDEV.S(A1:A6)
Spread: Coefficient of variation
- Looking at the calculations below, which is more variable, Carbon or Nitrogen?
- sd(c(48, 56, 78, 86, 90, 271))
[1] 79.2604[1] 84.10661 - =(STDEV.S(A1:A6)/AVERAGE(A1:A6))*100
Categorical data - table
- Different types with examples:
- Binary: We spray insects and see how many die
- Nominal: We count how animals, and their species, are in a forest
- Ordinal: Different disease levels for a plant, no disease, moderate, severe
- We can count the number observations belonging to each class, called frequency, f.
- Can present as a frequency table
Categorical data - Bar chart
- We then plot the table in ggplot
Categorical data - Bar chart
- Using
tidyverse
Categorical data - Bar chart
- NOTE: Bar charts should generally not used for continuous numerical data
Source: Weissgerber at al. (2015)
Numerical Data - Strip chart
- Often if we have a small data set (1-5 data points), we can use a stripchart to visualise our data. We will demonstrate using our soil carbon data set.
- What do we notice from the plot?
Numerical Data - Boxplot
- We can overlay our strip chart with a boxplot. This shows use the min/max, quartiles and median and can also show outliers.
- We generally use boxplots when we have more than 5 data points.
- See lecture notes for creating boxplot by hand.
Source: Nicholas (1999)
Numerical Data - Boxplot
- The follow produces a boxplot and also includes the jittered data points (red coloured)
- Note that there is on outlier which is the black data pointat the top of the plot
Numerical Data - Histogram
- Based on frequency table
- Height of each bar proportional to frequency - need to group data
- We can use histograms to describe the shape of distributions for continuous data sets that are larger than 20 data points.
Summary
- The following is a rough guide for plotting continuous data
- Remember for categorical data we use Tables and Bar Charts
Numerical Data - Symmetry
- For larger data sets, we can look at the mean, median and the histogram to determine if it is symmetrical.
[1] 93.8625[1] 93